Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: type safe metric labels #6201

Merged
merged 13 commits into from
Jan 2, 2024
Merged

Conversation

nflaig
Copy link
Member

@nflaig nflaig commented Dec 18, 2023

Motivation

Based on changes done in #6145 and #6143 I noticed a few limitations with in our current metrics and how we maintain labels

  • Labels are not self documenting, label values are added as comments which will likely become stale in the future
  • Labels values are not type safe
  • Labels can be omitted when setting value
  • Metric definition is not type safe, i.e. generic type can define label but labelNames can be omitted

Another issue is that we duplicate metric types a lot, not just in this codebase but also in libraries like discv5 or libp2p-gossipsub. The interface is quite opinionated and requires custom implementations (like avgMinMax or addCollect) at least in libraries we should not use those and keep it minimal. While this PR does not address the interface issue, it splits them into 3 separate types and cleans up some unused code introduced in #2312 which should make it easier to completely drop custom features like addCollect as well and just override collect directly as we do in discv5 service.ts#L197-L200. If we want to further push this could be addressed in a follow up, first need to ensure there are no metrics that require multiple collectors, however there is a risk of overriding collect twice which is a disadvantage compared to current addCollect behavior.

This is an initial push towards making our metrics more consumer friendly as noted in ChainSafe/js-libp2p-gossipsub#461.

Description

  • Move metric types to @lodestar/utils (similar to logger types)
  • Type safe label definition for both label name and value
  • Create enums if metric label have a predefined value set (self documenting)
  • Ensure labels are set if required by metric definition
  • Dropped unused child metrics
  • Removed duplicate gossip validation metrics
  • Update prom-client to v15.1.0 to ensure types are compatible with latest version (#6230)

Most of the changes are related to stricter label types. Type definitions require a more detailed review 👇

export interface Gauge<Labels extends LabelsGeneric = NoLabels> {
startTimer(): NoLabels extends Labels ? () => number : (labels: Labels) => number;
startTimer(labels: NoLabels extends Labels ? undefined : Labels): (labels?: Labels) => number;
inc: NoLabels extends Labels ? (value?: number) => void : (labels: Labels, value?: number) => void;
dec: NoLabels extends Labels ? (value?: number) => void : (labels: Labels, value?: number) => void;
set: NoLabels extends Labels ? (value: number) => void : (labels: Labels, value: number) => void;

Further considerations

We should find a way to avoid duplicating metric types in other libraries like discv5 or libp2p-gossipsub. While these could use @lodestar/utils, we might also consider having a separate / minimal package for just metrics (e.g. @lodestar/metrics) which includes all the types and some custom code like RegistryMetricCreator which would at least give other consumers an easy way to create the registry if they wanna use metrics our predefined metrics.

We could then also look into eventually support more interfaces like OTLP metrics, ChainSafe/js-libp2p-gossipsub#461 (comment). I haven't looked into the difference to make this work but might be easy to just implement a shim for it.

@@ -33,48 +20,3 @@ export class GaugeExtra<T extends string> extends Gauge<T> implements IGauge {
}
}
}

export class GaugeChild<T extends string> implements IGauge {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped unused child metrics to get closer to default prom-client interface

* - Add multiple collect functions after instantiation
* - Create child histograms with fixed labels
*/
export class HistogramExtra<T extends string> extends Histogram<T> implements IHistogram {
Copy link
Member Author

@nflaig nflaig Dec 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All this did was add child metric but this should not be required (same as GaugeExtra child)

name: "beacon_reqresp_rate_limiter_errors_total",
help: "Count rate limiter errors",
labelNames: ["method"],
}),
},

gossipValidationAccept: register.gauge<"topic">({
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped duplicate metrics, see previous related fix #6120

@@ -56,7 +64,7 @@ const discv5 = Discv5.create({
ip6: workerData.bindAddrs.ip6 ? multiaddr(workerData.bindAddrs.ip6) : undefined,
},
config: workerData.config,
metricsRegistry,
metricsRegistry: metricsRegistry as IDiscv5CreateOptions["metricsRegistry"],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discv5 metrics require collect function to be defined, see discv5/src/metrics.ts#L17. Can clean this up once we reuse same metric types across all packages

@nflaig nflaig marked this pull request as ready for review December 18, 2023 17:22
@nflaig nflaig requested a review from a team as a code owner December 18, 2023 17:22
Copy link

codecov bot commented Dec 18, 2023

Codecov Report

Merging #6201 (ae8b6d0) into unstable (21851b2) will decrease coverage by 0.06%.
The diff coverage is n/a.

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #6201      +/-   ##
============================================
- Coverage     80.95%   80.89%   -0.06%     
============================================
  Files           185      185              
  Lines         17935    17880      -55     
  Branches       1078     1078              
============================================
- Hits          14519    14464      -55     
  Misses         3389     3389              
  Partials         27       27              

Copy link
Contributor

github-actions bot commented Dec 18, 2023

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite Current: 685b30c Previous: a573b92 Ratio
altair processEth1Data - 250000 vs - 7PWei normalcase 488.77 us/op 2.2905 ms/op 0.21
Full benchmark results
Benchmark suite Current: 685b30c Previous: a573b92 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 612.78 us/op 569.88 us/op 1.08
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 80.073 us/op 97.373 us/op 0.82
BLS verify - blst-native 1.3032 ms/op 1.3183 ms/op 0.99
BLS verifyMultipleSignatures 3 - blst-native 2.7029 ms/op 2.7655 ms/op 0.98
BLS verifyMultipleSignatures 8 - blst-native 5.9624 ms/op 6.0616 ms/op 0.98
BLS verifyMultipleSignatures 32 - blst-native 21.833 ms/op 22.234 ms/op 0.98
BLS verifyMultipleSignatures 64 - blst-native 42.909 ms/op 44.013 ms/op 0.97
BLS verifyMultipleSignatures 128 - blst-native 85.944 ms/op 86.956 ms/op 0.99
BLS deserializing 10000 signatures 919.36 ms/op 939.38 ms/op 0.98
BLS deserializing 100000 signatures 9.2685 s/op 9.5140 s/op 0.97
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.3436 ms/op 1.3425 ms/op 1.00
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.5924 ms/op 1.5223 ms/op 1.05
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.2783 ms/op 2.3629 ms/op 0.96
BLS verifyMultipleSignatures - same message - 64 - blst-native 4.6278 ms/op 4.5487 ms/op 1.02
BLS verifyMultipleSignatures - same message - 128 - blst-native 5.5262 ms/op 7.4112 ms/op 0.75
BLS aggregatePubkeys 32 - blst-native 25.348 us/op 28.061 us/op 0.90
BLS aggregatePubkeys 128 - blst-native 99.132 us/op 101.52 us/op 0.98
getAttestationsForBlock 51.601 ms/op 52.492 ms/op 0.98
getSlashingsAndExits - default max 170.23 us/op 301.62 us/op 0.56
getSlashingsAndExits - 2k 408.99 us/op 408.65 us/op 1.00
proposeBlockBody type=full, size=empty 5.3177 ms/op 5.4203 ms/op 0.98
isKnown best case - 1 super set check 340.00 ns/op 541.00 ns/op 0.63
isKnown normal case - 2 super set checks 289.00 ns/op 350.00 ns/op 0.83
isKnown worse case - 16 super set checks 291.00 ns/op 356.00 ns/op 0.82
CheckpointStateCache - add get delete 5.0410 us/op 5.9620 us/op 0.85
validate api signedAggregateAndProof - struct 2.7573 ms/op 2.8142 ms/op 0.98
validate gossip signedAggregateAndProof - struct 2.7558 ms/op 2.8427 ms/op 0.97
validate gossip attestation - vc 640000 1.3502 ms/op 1.3889 ms/op 0.97
batch validate gossip attestation - vc 640000 - chunk 32 163.03 us/op 168.92 us/op 0.97
batch validate gossip attestation - vc 640000 - chunk 64 137.53 us/op 147.69 us/op 0.93
batch validate gossip attestation - vc 640000 - chunk 128 134.64 us/op 139.21 us/op 0.97
batch validate gossip attestation - vc 640000 - chunk 256 128.17 us/op 130.14 us/op 0.98
pickEth1Vote - no votes 1.0943 ms/op 1.1896 ms/op 0.92
pickEth1Vote - max votes 9.4605 ms/op 9.2334 ms/op 1.02
pickEth1Vote - Eth1Data hashTreeRoot value x2048 19.188 ms/op 15.726 ms/op 1.22
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 32.846 ms/op 24.651 ms/op 1.33
pickEth1Vote - Eth1Data fastSerialize value x2048 550.07 us/op 586.53 us/op 0.94
pickEth1Vote - Eth1Data fastSerialize tree x2048 5.3382 ms/op 4.9317 ms/op 1.08
bytes32 toHexString 463.00 ns/op 509.00 ns/op 0.91
bytes32 Buffer.toString(hex) 275.00 ns/op 282.00 ns/op 0.98
bytes32 Buffer.toString(hex) from Uint8Array 410.00 ns/op 467.00 ns/op 0.88
bytes32 Buffer.toString(hex) + 0x 279.00 ns/op 290.00 ns/op 0.96
Object access 1 prop 0.15400 ns/op 0.15800 ns/op 0.97
Map access 1 prop 0.13400 ns/op 0.14500 ns/op 0.92
Object get x1000 7.2140 ns/op 7.6440 ns/op 0.94
Map get x1000 0.70500 ns/op 0.80200 ns/op 0.88
Object set x1000 47.498 ns/op 53.806 ns/op 0.88
Map set x1000 36.901 ns/op 41.964 ns/op 0.88
Return object 10000 times 0.22580 ns/op 0.24220 ns/op 0.93
Throw Error 10000 times 3.7826 us/op 3.9846 us/op 0.95
fastMsgIdFn sha256 / 200 bytes 3.1350 us/op 3.3130 us/op 0.95
fastMsgIdFn h32 xxhash / 200 bytes 258.00 ns/op 321.00 ns/op 0.80
fastMsgIdFn h64 xxhash / 200 bytes 326.00 ns/op 366.00 ns/op 0.89
fastMsgIdFn sha256 / 1000 bytes 10.974 us/op 11.639 us/op 0.94
fastMsgIdFn h32 xxhash / 1000 bytes 386.00 ns/op 443.00 ns/op 0.87
fastMsgIdFn h64 xxhash / 1000 bytes 393.00 ns/op 450.00 ns/op 0.87
fastMsgIdFn sha256 / 10000 bytes 99.003 us/op 105.45 us/op 0.94
fastMsgIdFn h32 xxhash / 10000 bytes 1.8060 us/op 1.9720 us/op 0.92
fastMsgIdFn h64 xxhash / 10000 bytes 1.2460 us/op 1.3680 us/op 0.91
send data - 1000 256B messages 17.499 ms/op 20.568 ms/op 0.85
send data - 1000 512B messages 24.858 ms/op 28.260 ms/op 0.88
send data - 1000 1024B messages 39.104 ms/op 41.192 ms/op 0.95
send data - 1000 1200B messages 35.852 ms/op 39.193 ms/op 0.91
send data - 1000 2048B messages 41.982 ms/op 43.609 ms/op 0.96
send data - 1000 4096B messages 41.172 ms/op 41.686 ms/op 0.99
send data - 1000 16384B messages 117.31 ms/op 119.99 ms/op 0.98
send data - 1000 65536B messages 447.48 ms/op 460.13 ms/op 0.97
enrSubnets - fastDeserialize 64 bits 1.2020 us/op 1.4160 us/op 0.85
enrSubnets - ssz BitVector 64 bits 409.00 ns/op 464.00 ns/op 0.88
enrSubnets - fastDeserialize 4 bits 161.00 ns/op 186.00 ns/op 0.87
enrSubnets - ssz BitVector 4 bits 411.00 ns/op 432.00 ns/op 0.95
prioritizePeers score -10:0 att 32-0.1 sync 2-0 106.56 us/op 111.68 us/op 0.95
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 120.01 us/op 137.22 us/op 0.87
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 154.34 us/op 182.55 us/op 0.85
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 275.25 us/op 327.55 us/op 0.84
prioritizePeers score 0:0 att 64-1 sync 4-1 315.95 us/op 367.56 us/op 0.86
array of 16000 items push then shift 1.5188 us/op 1.6673 us/op 0.91
LinkedList of 16000 items push then shift 8.4250 ns/op 9.3710 ns/op 0.90
array of 16000 items push then pop 83.888 ns/op 95.692 ns/op 0.88
LinkedList of 16000 items push then pop 8.3730 ns/op 8.8160 ns/op 0.95
array of 24000 items push then shift 2.3167 us/op 2.4210 us/op 0.96
LinkedList of 24000 items push then shift 8.6480 ns/op 9.1720 ns/op 0.94
array of 24000 items push then pop 96.903 ns/op 139.19 ns/op 0.70
LinkedList of 24000 items push then pop 8.5550 ns/op 8.7180 ns/op 0.98
intersect bitArray bitLen 8 6.4240 ns/op 6.4450 ns/op 1.00
intersect array and set length 8 61.564 ns/op 68.093 ns/op 0.90
intersect bitArray bitLen 128 33.588 ns/op 34.434 ns/op 0.98
intersect array and set length 128 850.65 ns/op 974.22 ns/op 0.87
bitArray.getTrueBitIndexes() bitLen 128 1.5070 us/op 1.6100 us/op 0.94
bitArray.getTrueBitIndexes() bitLen 248 2.4410 us/op 3.0420 us/op 0.80
bitArray.getTrueBitIndexes() bitLen 512 4.6510 us/op 5.2280 us/op 0.89
Buffer.concat 32 items 972.00 ns/op 1.0530 us/op 0.92
Uint8Array.set 32 items 2.5060 us/op 2.9280 us/op 0.86
Set add up to 64 items then delete first 4.1201 us/op 4.4680 us/op 0.92
OrderedSet add up to 64 items then delete first 5.2109 us/op 5.8232 us/op 0.89
Set add up to 64 items then delete last 4.4062 us/op 5.0080 us/op 0.88
OrderedSet add up to 64 items then delete last 5.7162 us/op 6.2607 us/op 0.91
Set add up to 64 items then delete middle 4.4215 us/op 5.2150 us/op 0.85
OrderedSet add up to 64 items then delete middle 7.1650 us/op 7.6162 us/op 0.94
Set add up to 128 items then delete first 9.0233 us/op 10.001 us/op 0.90
OrderedSet add up to 128 items then delete first 11.570 us/op 12.956 us/op 0.89
Set add up to 128 items then delete last 9.0554 us/op 10.347 us/op 0.88
OrderedSet add up to 128 items then delete last 11.964 us/op 13.708 us/op 0.87
Set add up to 128 items then delete middle 8.9350 us/op 10.662 us/op 0.84
OrderedSet add up to 128 items then delete middle 17.016 us/op 17.644 us/op 0.96
Set add up to 256 items then delete first 18.295 us/op 20.355 us/op 0.90
OrderedSet add up to 256 items then delete first 23.561 us/op 25.617 us/op 0.92
Set add up to 256 items then delete last 17.856 us/op 18.542 us/op 0.96
OrderedSet add up to 256 items then delete last 24.101 us/op 23.853 us/op 1.01
Set add up to 256 items then delete middle 17.791 us/op 18.303 us/op 0.97
OrderedSet add up to 256 items then delete middle 44.558 us/op 47.112 us/op 0.95
transfer serialized Status (84 B) 1.7380 us/op 1.7420 us/op 1.00
copy serialized Status (84 B) 1.4280 us/op 1.4810 us/op 0.96
transfer serialized SignedVoluntaryExit (112 B) 1.8520 us/op 1.8770 us/op 0.99
copy serialized SignedVoluntaryExit (112 B) 1.4770 us/op 1.5180 us/op 0.97
transfer serialized ProposerSlashing (416 B) 2.0930 us/op 2.0270 us/op 1.03
copy serialized ProposerSlashing (416 B) 2.5250 us/op 1.8510 us/op 1.36
transfer serialized Attestation (485 B) 2.4250 us/op 2.0480 us/op 1.18
copy serialized Attestation (485 B) 2.9370 us/op 1.8750 us/op 1.57
transfer serialized AttesterSlashing (33232 B) 3.1040 us/op 2.2190 us/op 1.40
copy serialized AttesterSlashing (33232 B) 7.4070 us/op 6.0950 us/op 1.22
transfer serialized Small SignedBeaconBlock (128000 B) 3.5620 us/op 2.7060 us/op 1.32
copy serialized Small SignedBeaconBlock (128000 B) 14.607 us/op 17.731 us/op 0.82
transfer serialized Avg SignedBeaconBlock (200000 B) 3.7090 us/op 3.0920 us/op 1.20
copy serialized Avg SignedBeaconBlock (200000 B) 20.682 us/op 27.432 us/op 0.75
transfer serialized BlobsSidecar (524380 B) 2.9690 us/op 3.0030 us/op 0.99
copy serialized BlobsSidecar (524380 B) 93.757 us/op 89.724 us/op 1.04
transfer serialized Big SignedBeaconBlock (1000000 B) 3.3270 us/op 3.0740 us/op 1.08
copy serialized Big SignedBeaconBlock (1000000 B) 149.42 us/op 208.84 us/op 0.72
pass gossip attestations to forkchoice per slot 4.2578 ms/op 4.4864 ms/op 0.95
forkChoice updateHead vc 100000 bc 64 eq 0 679.86 us/op 705.73 us/op 0.96
forkChoice updateHead vc 600000 bc 64 eq 0 4.2105 ms/op 4.4359 ms/op 0.95
forkChoice updateHead vc 1000000 bc 64 eq 0 6.9520 ms/op 7.4063 ms/op 0.94
forkChoice updateHead vc 600000 bc 320 eq 0 4.1505 ms/op 4.1485 ms/op 1.00
forkChoice updateHead vc 600000 bc 1200 eq 0 4.2926 ms/op 4.2844 ms/op 1.00
forkChoice updateHead vc 600000 bc 7200 eq 0 5.2125 ms/op 5.4257 ms/op 0.96
forkChoice updateHead vc 600000 bc 64 eq 1000 11.007 ms/op 11.375 ms/op 0.97
forkChoice updateHead vc 600000 bc 64 eq 10000 11.680 ms/op 12.191 ms/op 0.96
forkChoice updateHead vc 600000 bc 64 eq 300000 15.316 ms/op 17.353 ms/op 0.88
computeDeltas 500000 validators 300 proto nodes 6.6669 ms/op 6.6254 ms/op 1.01
computeDeltas 500000 validators 1200 proto nodes 6.4044 ms/op 6.4975 ms/op 0.99
computeDeltas 500000 validators 7200 proto nodes 6.3282 ms/op 6.4665 ms/op 0.98
computeDeltas 750000 validators 300 proto nodes 9.5496 ms/op 9.7886 ms/op 0.98
computeDeltas 750000 validators 1200 proto nodes 9.7218 ms/op 9.4271 ms/op 1.03
computeDeltas 750000 validators 7200 proto nodes 9.7967 ms/op 9.7401 ms/op 1.01
computeDeltas 1400000 validators 300 proto nodes 18.030 ms/op 18.654 ms/op 0.97
computeDeltas 1400000 validators 1200 proto nodes 19.287 ms/op 19.300 ms/op 1.00
computeDeltas 1400000 validators 7200 proto nodes 18.768 ms/op 19.209 ms/op 0.98
computeDeltas 2100000 validators 300 proto nodes 27.568 ms/op 27.465 ms/op 1.00
computeDeltas 2100000 validators 1200 proto nodes 28.740 ms/op 28.110 ms/op 1.02
computeDeltas 2100000 validators 7200 proto nodes 28.889 ms/op 27.744 ms/op 1.04
computeProposerBoostScoreFromBalances 500000 validators 3.6751 ms/op 3.7654 ms/op 0.98
computeProposerBoostScoreFromBalances 750000 validators 3.6170 ms/op 3.7069 ms/op 0.98
computeProposerBoostScoreFromBalances 1400000 validators 3.6595 ms/op 3.7674 ms/op 0.97
computeProposerBoostScoreFromBalances 2100000 validators 3.5975 ms/op 3.7611 ms/op 0.96
altair processAttestation - 250000 vs - 7PWei normalcase 2.2538 ms/op 2.3588 ms/op 0.96
altair processAttestation - 250000 vs - 7PWei worstcase 3.2956 ms/op 3.2895 ms/op 1.00
altair processAttestation - setStatus - 1/6 committees join 142.94 us/op 145.79 us/op 0.98
altair processAttestation - setStatus - 1/3 committees join 273.28 us/op 287.50 us/op 0.95
altair processAttestation - setStatus - 1/2 committees join 393.32 us/op 387.29 us/op 1.02
altair processAttestation - setStatus - 2/3 committees join 470.04 us/op 486.07 us/op 0.97
altair processAttestation - setStatus - 4/5 committees join 657.17 us/op 686.13 us/op 0.96
altair processAttestation - setStatus - 100% committees join 789.52 us/op 792.61 us/op 1.00
altair processBlock - 250000 vs - 7PWei normalcase 11.307 ms/op 11.098 ms/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase hashState 39.646 ms/op 38.275 ms/op 1.04
altair processBlock - 250000 vs - 7PWei worstcase 41.089 ms/op 39.286 ms/op 1.05
altair processBlock - 250000 vs - 7PWei worstcase hashState 97.013 ms/op 94.246 ms/op 1.03
phase0 processBlock - 250000 vs - 7PWei normalcase 3.4456 ms/op 2.5307 ms/op 1.36
phase0 processBlock - 250000 vs - 7PWei worstcase 32.782 ms/op 31.068 ms/op 1.06
altair processEth1Data - 250000 vs - 7PWei normalcase 488.77 us/op 2.2905 ms/op 0.21
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 7.0960 us/op 13.683 us/op 0.52
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 96.967 us/op 73.955 us/op 1.31
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 25.720 us/op 21.236 us/op 1.21
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 6.9940 us/op 8.0780 us/op 0.87
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 205.89 us/op 196.94 us/op 1.05
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.0168 ms/op 1.3288 ms/op 0.77
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 2.7955 ms/op 2.0042 ms/op 1.39
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 2.7668 ms/op 1.5548 ms/op 1.78
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 3.3298 ms/op 4.2498 ms/op 0.78
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 2.0621 ms/op 2.4872 ms/op 0.83
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 5.2283 ms/op 5.1395 ms/op 1.02
Tree 40 250000 create 333.17 ms/op 372.63 ms/op 0.89
Tree 40 250000 get(125000) 188.55 ns/op 207.54 ns/op 0.91
Tree 40 250000 set(125000) 970.54 ns/op 1.0446 us/op 0.93
Tree 40 250000 toArray() 18.468 ms/op 19.763 ms/op 0.93
Tree 40 250000 iterate all - toArray() + loop 18.761 ms/op 22.287 ms/op 0.84
Tree 40 250000 iterate all - get(i) 63.329 ms/op 70.759 ms/op 0.89
MutableVector 250000 create 19.984 ms/op 14.189 ms/op 1.41
MutableVector 250000 get(125000) 6.5790 ns/op 6.5090 ns/op 1.01
MutableVector 250000 set(125000) 260.04 ns/op 294.44 ns/op 0.88
MutableVector 250000 toArray() 3.1660 ms/op 4.5655 ms/op 0.69
MutableVector 250000 iterate all - toArray() + loop 3.2630 ms/op 4.5273 ms/op 0.72
MutableVector 250000 iterate all - get(i) 1.4738 ms/op 1.6001 ms/op 0.92
Array 250000 create 2.7195 ms/op 3.7108 ms/op 0.73
Array 250000 clone - spread 1.2209 ms/op 1.2741 ms/op 0.96
Array 250000 get(125000) 0.99200 ns/op 1.0630 ns/op 0.93
Array 250000 set(125000) 3.9130 ns/op 4.2580 ns/op 0.92
Array 250000 iterate all - loop 156.17 us/op 173.52 us/op 0.90
effectiveBalanceIncrements clone Uint8Array 300000 25.958 us/op 30.980 us/op 0.84
effectiveBalanceIncrements clone MutableVector 300000 352.00 ns/op 359.00 ns/op 0.98
effectiveBalanceIncrements rw all Uint8Array 300000 188.27 us/op 203.36 us/op 0.93
effectiveBalanceIncrements rw all MutableVector 300000 82.330 ms/op 79.271 ms/op 1.04
phase0 afterProcessEpoch - 250000 vs - 7PWei 104.96 ms/op 113.33 ms/op 0.93
phase0 beforeProcessEpoch - 250000 vs - 7PWei 58.854 ms/op 51.709 ms/op 1.14
altair processEpoch - mainnet_e81889 561.24 ms/op 508.87 ms/op 1.10
mainnet_e81889 - altair beforeProcessEpoch 85.011 ms/op 82.448 ms/op 1.03
mainnet_e81889 - altair processJustificationAndFinalization 25.304 us/op 14.068 us/op 1.80
mainnet_e81889 - altair processInactivityUpdates 6.1001 ms/op 6.0199 ms/op 1.01
mainnet_e81889 - altair processRewardsAndPenalties 77.427 ms/op 64.212 ms/op 1.21
mainnet_e81889 - altair processRegistryUpdates 4.0270 us/op 2.5380 us/op 1.59
mainnet_e81889 - altair processSlashings 655.00 ns/op 504.00 ns/op 1.30
mainnet_e81889 - altair processEth1DataReset 782.00 ns/op 460.00 ns/op 1.70
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.6550 ms/op 1.4277 ms/op 1.86
mainnet_e81889 - altair processSlashingsReset 5.7440 us/op 3.4340 us/op 1.67
mainnet_e81889 - altair processRandaoMixesReset 6.5290 us/op 4.2230 us/op 1.55
mainnet_e81889 - altair processHistoricalRootsUpdate 1.1640 us/op 682.00 ns/op 1.71
mainnet_e81889 - altair processParticipationFlagUpdates 3.1600 us/op 1.9570 us/op 1.61
mainnet_e81889 - altair processSyncCommitteeUpdates 734.00 ns/op 1.2970 us/op 0.57
mainnet_e81889 - altair afterProcessEpoch 112.72 ms/op 118.71 ms/op 0.95
capella processEpoch - mainnet_e217614 2.3445 s/op 2.1016 s/op 1.12
mainnet_e217614 - capella beforeProcessEpoch 490.15 ms/op 495.10 ms/op 0.99
mainnet_e217614 - capella processJustificationAndFinalization 25.538 us/op 14.889 us/op 1.72
mainnet_e217614 - capella processInactivityUpdates 20.551 ms/op 20.813 ms/op 0.99
mainnet_e217614 - capella processRewardsAndPenalties 427.92 ms/op 421.04 ms/op 1.02
mainnet_e217614 - capella processRegistryUpdates 25.288 us/op 21.241 us/op 1.19
mainnet_e217614 - capella processSlashings 818.00 ns/op 569.00 ns/op 1.44
mainnet_e217614 - capella processEth1DataReset 765.00 ns/op 438.00 ns/op 1.75
mainnet_e217614 - capella processEffectiveBalanceUpdates 4.1115 ms/op 4.1553 ms/op 0.99
mainnet_e217614 - capella processSlashingsReset 4.7310 us/op 3.3590 us/op 1.41
mainnet_e217614 - capella processRandaoMixesReset 7.3250 us/op 5.2720 us/op 1.39
mainnet_e217614 - capella processHistoricalRootsUpdate 875.00 ns/op 574.00 ns/op 1.52
mainnet_e217614 - capella processParticipationFlagUpdates 3.2330 us/op 2.1580 us/op 1.50
mainnet_e217614 - capella afterProcessEpoch 319.88 ms/op 321.80 ms/op 0.99
phase0 processEpoch - mainnet_e58758 510.68 ms/op 409.14 ms/op 1.25
mainnet_e58758 - phase0 beforeProcessEpoch 155.43 ms/op 111.59 ms/op 1.39
mainnet_e58758 - phase0 processJustificationAndFinalization 20.540 us/op 16.402 us/op 1.25
mainnet_e58758 - phase0 processRewardsAndPenalties 66.069 ms/op 55.535 ms/op 1.19
mainnet_e58758 - phase0 processRegistryUpdates 14.195 us/op 8.9690 us/op 1.58
mainnet_e58758 - phase0 processSlashings 859.00 ns/op 525.00 ns/op 1.64
mainnet_e58758 - phase0 processEth1DataReset 648.00 ns/op 381.00 ns/op 1.70
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.1633 ms/op 1.1288 ms/op 1.92
mainnet_e58758 - phase0 processSlashingsReset 4.4740 us/op 3.0370 us/op 1.47
mainnet_e58758 - phase0 processRandaoMixesReset 5.9850 us/op 4.1120 us/op 1.46
mainnet_e58758 - phase0 processHistoricalRootsUpdate 962.00 ns/op 408.00 ns/op 2.36
mainnet_e58758 - phase0 processParticipationRecordUpdates 5.6900 us/op 3.7310 us/op 1.53
mainnet_e58758 - phase0 afterProcessEpoch 92.309 ms/op 96.141 ms/op 0.96
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.5064 ms/op 1.3618 ms/op 1.84
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4211 ms/op 1.4536 ms/op 0.98
altair processInactivityUpdates - 250000 normalcase 30.797 ms/op 26.291 ms/op 1.17
altair processInactivityUpdates - 250000 worstcase 34.750 ms/op 21.037 ms/op 1.65
phase0 processRegistryUpdates - 250000 normalcase 13.428 us/op 8.4380 us/op 1.59
phase0 processRegistryUpdates - 250000 badcase_full_deposits 532.66 us/op 335.46 us/op 1.59
phase0 processRegistryUpdates - 250000 worstcase 0.5 127.85 ms/op 139.02 ms/op 0.92
altair processRewardsAndPenalties - 250000 normalcase 79.263 ms/op 55.279 ms/op 1.43
altair processRewardsAndPenalties - 250000 worstcase 70.651 ms/op 57.016 ms/op 1.24
phase0 getAttestationDeltas - 250000 normalcase 9.3067 ms/op 8.7592 ms/op 1.06
phase0 getAttestationDeltas - 250000 worstcase 8.8254 ms/op 8.8759 ms/op 0.99
phase0 processSlashings - 250000 worstcase 128.47 us/op 81.217 us/op 1.58
altair processSyncCommitteeUpdates - 250000 147.75 ms/op 154.61 ms/op 0.96
BeaconState.hashTreeRoot - No change 286.00 ns/op 235.00 ns/op 1.22
BeaconState.hashTreeRoot - 1 full validator 141.21 us/op 144.14 us/op 0.98
BeaconState.hashTreeRoot - 32 full validator 1.7258 ms/op 1.7182 ms/op 1.00
BeaconState.hashTreeRoot - 512 full validator 17.733 ms/op 17.753 ms/op 1.00
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 208.19 us/op 188.77 us/op 1.10
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.1255 ms/op 2.0371 ms/op 1.04
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 31.100 ms/op 32.511 ms/op 0.96
BeaconState.hashTreeRoot - 1 balances 176.94 us/op 131.57 us/op 1.34
BeaconState.hashTreeRoot - 32 balances 1.5775 ms/op 1.1833 ms/op 1.33
BeaconState.hashTreeRoot - 512 balances 15.500 ms/op 12.551 ms/op 1.24
BeaconState.hashTreeRoot - 250000 balances 240.87 ms/op 198.64 ms/op 1.21
aggregationBits - 2048 els - zipIndexesInBitList 16.081 us/op 16.896 us/op 0.95
byteArrayEquals 32 70.871 ns/op 72.981 ns/op 0.97
Buffer.compare 32 52.228 ns/op 54.436 ns/op 0.96
byteArrayEquals 1024 1.9323 us/op 1.9920 us/op 0.97
Buffer.compare 1024 68.085 ns/op 69.102 ns/op 0.99
byteArrayEquals 16384 30.744 us/op 31.694 us/op 0.97
Buffer.compare 16384 244.15 ns/op 253.23 ns/op 0.96
byteArrayEquals 123687377 241.79 ms/op 241.47 ms/op 1.00
Buffer.compare 123687377 6.2131 ms/op 6.0414 ms/op 1.03
byteArrayEquals 32 - diff last byte 69.853 ns/op 70.457 ns/op 0.99
Buffer.compare 32 - diff last byte 54.520 ns/op 55.419 ns/op 0.98
byteArrayEquals 1024 - diff last byte 1.9846 us/op 2.0102 us/op 0.99
Buffer.compare 1024 - diff last byte 69.773 ns/op 71.127 ns/op 0.98
byteArrayEquals 16384 - diff last byte 31.551 us/op 32.084 us/op 0.98
Buffer.compare 16384 - diff last byte 244.91 ns/op 263.82 ns/op 0.93
byteArrayEquals 123687377 - diff last byte 238.08 ms/op 244.34 ms/op 0.97
Buffer.compare 123687377 - diff last byte 6.0936 ms/op 6.0693 ms/op 1.00
byteArrayEquals 32 - random bytes 5.3750 ns/op 5.3680 ns/op 1.00
Buffer.compare 32 - random bytes 59.949 ns/op 60.135 ns/op 1.00
byteArrayEquals 1024 - random bytes 5.1270 ns/op 5.1270 ns/op 1.00
Buffer.compare 1024 - random bytes 59.108 ns/op 59.531 ns/op 0.99
byteArrayEquals 16384 - random bytes 5.1500 ns/op 5.1290 ns/op 1.00
Buffer.compare 16384 - random bytes 59.235 ns/op 59.570 ns/op 0.99
byteArrayEquals 123687377 - random bytes 8.3900 ns/op 8.2200 ns/op 1.02
Buffer.compare 123687377 - random bytes 62.490 ns/op 62.700 ns/op 1.00
regular array get 100000 times 43.540 us/op 43.849 us/op 0.99
wrappedArray get 100000 times 43.887 us/op 43.798 us/op 1.00
arrayWithProxy get 100000 times 14.305 ms/op 14.018 ms/op 1.02
ssz.Root.equals 53.012 ns/op 53.198 ns/op 1.00
byteArrayEquals 52.375 ns/op 52.418 ns/op 1.00
Buffer.compare 10.653 ns/op 10.790 ns/op 0.99
shuffle list - 16384 els 6.9165 ms/op 6.8452 ms/op 1.01
shuffle list - 250000 els 102.20 ms/op 100.59 ms/op 1.02
processSlot - 1 slots 17.112 us/op 16.569 us/op 1.03
processSlot - 32 slots 3.9428 ms/op 3.0915 ms/op 1.28
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 66.449 ms/op 58.115 ms/op 1.14
getCommitteeAssignments - req 1 vs - 250000 vc 2.4467 ms/op 2.4590 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 3.5994 ms/op 3.6279 ms/op 0.99
getCommitteeAssignments - req 1000 vs - 250000 vc 3.9590 ms/op 3.9492 ms/op 1.00
findModifiedValidators - 10000 modified validators 522.49 ms/op 526.14 ms/op 0.99
findModifiedValidators - 1000 modified validators 433.26 ms/op 406.59 ms/op 1.07
findModifiedValidators - 100 modified validators 391.39 ms/op 423.96 ms/op 0.92
findModifiedValidators - 10 modified validators 419.39 ms/op 402.55 ms/op 1.04
findModifiedValidators - 1 modified validators 408.53 ms/op 397.47 ms/op 1.03
findModifiedValidators - no difference 412.46 ms/op 374.68 ms/op 1.10
compare ViewDUs 5.0303 s/op 4.2899 s/op 1.17
compare each validator Uint8Array 1.9882 s/op 1.6871 s/op 1.18
compare ViewDU to Uint8Array 1.0720 s/op 1.2342 s/op 0.87
migrate state 1000000 validators, 24 modified, 0 new 880.57 ms/op 824.13 ms/op 1.07
migrate state 1000000 validators, 1700 modified, 1000 new 1.2089 s/op 1.0539 s/op 1.15
migrate state 1000000 validators, 3400 modified, 2000 new 1.4208 s/op 1.2825 s/op 1.11
migrate state 1500000 validators, 24 modified, 0 new 897.67 ms/op 756.61 ms/op 1.19
migrate state 1500000 validators, 1700 modified, 1000 new 1.2000 s/op 1.0864 s/op 1.10
migrate state 1500000 validators, 3400 modified, 2000 new 1.4541 s/op 1.3212 s/op 1.10
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.2700 ns/op 4.1500 ns/op 1.03
state getBlockRootAtSlot - 250000 vs - 7PWei 890.24 ns/op 742.98 ns/op 1.20
computeProposers - vc 250000 9.5889 ms/op 9.3798 ms/op 1.02
computeEpochShuffling - vc 250000 102.58 ms/op 101.72 ms/op 1.01
getNextSyncCommittee - vc 250000 153.63 ms/op 153.65 ms/op 1.00
computeSigningRoot for AttestationData 27.372 us/op 25.565 us/op 1.07
hash AttestationData serialized data then Buffer.toString(base64) 2.2436 us/op 2.2502 us/op 1.00
toHexString serialized data 1.0217 us/op 1.0326 us/op 0.99
Buffer.toString(base64) 209.78 ns/op 204.27 ns/op 1.03

by benchmarkbot/action

dec: NoLabels extends Labels ? (value?: number) => void : (labels: Labels, value?: number) => void;
set: NoLabels extends Labels ? (value: number) => void : (labels: Labels, value: number) => void;

collect?(): void;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not we add deprecated warning here as we did for GuageExtra?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overriding collect is the way to go if you use default prom-client Gauge. If we use the custom GaugeExtra on the other hand we should use addCollect instead. It would be good if we move away from GaugeExtra in the long-term and only use prom-client default metric classes.

I completely removed the collect method from GaugeExtra now as there is no reason to use it there.

see commit 2e7d70b

packages/utils/src/metrics.ts Show resolved Hide resolved
packages/utils/src/metrics.ts Outdated Show resolved Hide resolved
startTimer(): NoLabels extends Labels ? () => number : (labels: Labels) => number;
startTimer(labels: NoLabels extends Labels ? undefined : Labels): (labels?: Labels) => number;

inc: NoLabels extends Labels ? (value?: number) => void : (labels: Labels, value?: number) => void;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't get that one. If we have the Labels type with multiple keys, do we have to increment every label the same time, else this type will raise error.

Copy link
Member Author

@nflaig nflaig Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have to increment every label the same time, else this type will raise error.

I am not sure what you mean by "increment every label at the same time". What this does is that if you metric has a label defined, then when setting a value for that metric it must specify a label.

Let's look at an example

requestErrors: register.gauge<{routeId: string}>({
  name: "vc_rest_api_client_request_errors_total",
  help: "Total count of errors on REST API client requests by routeId",
  labelNames: ["routeId"],
}),

This is no longer allowed

// An argument for 'labels' was not provided.
this.metrics?.requestErrors.inc();

You are forced to do

this.metrics?.requestErrors.inc({routeId});

I don't see any reason why you would want to set this metric without a label, it would mean there is a unlabeled value which could be combined with further unlabeled values of the same metric but this defeats the purpose of adding a label in the first place, to label and separate values of the same metric.

I only found a few cases were we forgot to set the label but fixed those in this PR as well.

Note that there is also still the option to make the label optional when defining the type for it, I don't think there is any reason to do this but it's an option.

packages/utils/src/metrics.ts Outdated Show resolved Hide resolved
packages/utils/src/metrics.ts Show resolved Hide resolved
export type GaugeConfig<Labels extends LabelsGeneric> = {
name: string;
help: string;
} & (NoLabels extends Labels ? {labelNames?: never} : {labelNames: [LabelKeys<Labels>, ...LabelKeys<Labels>[]]});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This type is same as:

Suggested change
} & (NoLabels extends Labels ? {labelNames?: never} : {labelNames: [LabelKeys<Labels>, ...LabelKeys<Labels>[]]});
} & (NoLabels extends Labels ? {labelNames?: never} : {labelNames: LabelKeys<Labels>[]});

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The behavior is different

labelNames: LabelKeys<Labels>[] allows empty arrays

blockProductionTime: register.histogram<{source: ProducedBlockSource}>({
  // ...
  labelNames: [], // No complaints
}),

whereas labelNames: [LabelKeys<Labels>, ...LabelKeys<Labels>[]] requires you to actually add the label name

blockProductionTime: register.histogram<{source: ProducedBlockSource}>({
  // ...
  labelNames: [], // Type '[]' is not assignable to type '["source", ..."source"[]]'.
}),

The ...LabelKeys<Labels>[] part is required for cases were we have multiple labels

Comment on lines 68 to 70
gauge<Labels extends LabelsGeneric>(config: GaugeConfig<Labels>): Gauge<Labels>;
histogram<Labels extends LabelsGeneric>(config: HistogramConfig<Labels>): Histogram<Labels>;
counter<Labels extends LabelsGeneric>(config: CounterConfig<Labels>): Counter<Labels>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we not set default value for generics here, else we have to specify type everywhere we use the register.

Copy link
Member Author

@nflaig nflaig Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't have to specify the type explicitly, it is inferred from the configuration

e.g. see histogram usage without labels

parentBlockDistance: register.histogram({

But I added them now to address the previous issue regarding usage of {} type #6201 (comment)

see commit 2a14a4b

Copy link
Member

@wemeetagain wemeetagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice, looks like it caught a few cases of us missing labels.

I think eventually we should move the metrics utils out of the monorepo so it can be used elsewhere without being pegged to the lodestar release cadence.

@wemeetagain wemeetagain merged commit b7925f6 into unstable Jan 2, 2024
15 checks passed
@wemeetagain wemeetagain deleted the nflaig/type-safe-metric-labels branch January 2, 2024 14:56
@nflaig
Copy link
Member Author

nflaig commented Jan 2, 2024

I think eventually we should move the metrics utils out of the monorepo

That's a good idea and better than having a lodestar/metrics package but I am not sure if a separate package is worth it right now as it would basically just have type definitions and the metrics factory. If we would like to support other metrics like OTLP then it could make sense to have a package that also includes a shim for it.

Next step would be to align metric types in other packages and then it should be quite easy to update those once we have a separate metrics package.

@wemeetagain
Copy link
Member

🎉 This PR is included in v1.14.0 🎉

ensi321 pushed a commit to ensi321/lodestar that referenced this pull request Jan 22, 2024
* refactor: type safe metric labels

* Update metrics after merging unstable

* Remove collect method from GaugeExtra

* Remove startTimer method from Gauge interface

* Default to NoLabels to fix type issues

* Allow to partially set labels in startTimer

* Fix type compatibility with prom-client Histogram

* Sort metric types

* chore: update prom-client to v15.1.0 (ChainSafe#6230)

* Add metric type tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants